Usenet 1993 July/InfoMagic USENET CD-ROM July 1993.ISO/sources/std_unix/archive/text0013.txt

Text File | 1993-07-06 | 3.6 KB | 88 lines

Submitted-by: [e-mail] (Dave Cline)

In article <[e-mail]>,
    [e-mail] (Jeffrey Kegler) writes:

>Here's the problem.  In the absence of a test method standard, the
>implementor of the base standard must come as close as possible to the
>original standard.

To be a bit more precise, in the absence of a conformance test suite,
the implementor must *say* they implement the base standard.  It's
not quite the same thing in the real world.

>If a test method standard exists, however, he need
>only implement to pass the tests in that standard.

Here, as in the following, no clear distinction is made between
a test method standard and a conformance test suite is made.

Conformance test suites existed long before anyone thought of
writing a test method standard.  Almost all of the arguments that
are advanced here against test method standards apply equally to
conformance test suites.

>Standardized test methods imply, perhaps paradoxically, a considerably
>less rigorous standard than their original base standard.  Almost all
>standards of POSIX complexity will contain requirements which are not
>practically testable.  The standardized test method, in effect,
>repeals these.
>
>The mathematical argument that the base standard implied by the test
>method standard must be less rigorous than the original is
>straightforward.  Most POSIX standards will specify a sufficiently
>complex system to implement a Turing machine.  Testing that all
>programs for a Turing machine correctly execute on a given black box
>is impossible, even in theory -- it implies the solution of whole sets
>of undecidable problems.

I think that the reference to Turing might have been intended to refer
to Goedel, but anyway, the reference is misplaced.   A Turing machine
requires infinite storage.  If only finite storage is present, then
one can, in theory, enumerate all the possible programs and their
results.  With finite storage there is no halting problem.  One may
well choose not to wait for the answers, though.  :-)

There are hard problems.  There are complex standards.  Exhaustive
black box testing of standards is not possible for many different
reasons.

All of these statements have nothing whatever to do with Turing machines.

Any black box test such as a conformance test suite samples only
a small portion of the potential state space of a large system.
As with any test, it can only show the presence of bugs.

We certainly should not criticize black box conformance testing
because it cannot show the absence of bugs.

>It would seem to me to be prudent to produce test method standards on
>a limited, trial basis, if at all.

It seems valid to argue against formal test method standards because:

  a) they are hard to write.
  b) they are just as error-prone as the original standard.
  c) there is no mechanical way to ensure that both standards
     say the same thing.
  d) they are not cost-effective.
  e) the audience for such standards is limited.
  f) they are not technology enablers.  Conformance test suites
     will be written even if test method standards are not.
  g) the time required to create a test method standard impacts
     the timely release of the base standards.

I agree with the above statements.

However, one should distinguish between test method standards
and conformance test suites.  The conformance test suites will
end up being written whether a test method standard exists
or not (e.g. languages have not been "blessed" with test method
standards, but language conformance testing is accepted industry
practice).

--

Volume-Number: Volume 31, Number 14